Skip to main content

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Nemo_bis Date: Nov 3, 2013 3:23am
Forum: web Subject: Re: Allow IA to index, prevent search engines

On a standard website, the easiest approach would be to put those you want to be archived in a different directory, use robots.txt instead of meta tags and whitelist the Wayback machine for that directory (though I don't know if Allow: is accepted by the machine), User-agent: ia_archiver according to